OCCASIONAL PAPER 




Measures of student success: 
can we predict module- 
completion rates? 

Jean e^e Learned 
Western Sydney Institute ofTAFE 



Participant in the NCVER Building Researcher Capacity 
Community of Practice Scholarship Program 2009 





Australian Government 

Department of Education, Employment 
and Workplace Relations 












NCVER 



Measures of student success: 

can we predict module-completion rates? 

JEANETTE LEARNED 

Western Sydney institute of TAFE 



Participant in the NCVER Building 
Researcher Capacity Community of 
Practice Scholarship Program 2009 




Australian Government 

Department of Education, Employment 
and Workplace Relations 



The views and opinions expressed in this document are those of the author/ project team and do 
not necessarily reflect the views of the Australian Government or state and territory governments. 
Any interpretation of data is the responsibility of the author/project team 



As part of the National Centre for Vocational Education Research (NCVER) Building 
Researcher Capacity Scheme, a community of practice scholarship program has been 
created to encourage a culture of research in vocational education and training (VET) 
organisations. With the guidance of an experienced mentor, VET practitioners without 
any formal research experience undertake their own work-based research project. The 
scholarships also provide participants with an opportunity to have their research peer- 
reviewed and published by NCVER. 

For more information see: <http:/ /www.ncver.edu.au/research/opportunities.html# 
Community_of_practice_scholarships_for_VET_practidoners>. 



© Commonwealth of Australia, 2010 

This work has been produced by the National Centre for Vocational Education Research (NCVER) 
under the National Vocational Education and Training Research and Evaluation (NVETRE) 
Program, which is coordinated and managed by NCVER on behalf of the Australian Government 
and state and territory governments. Funding is provided through the Department of Education, 
Employment and Workplace Relations. Apart from any use permitted under the Copyright A.ct 1968, 
no part of this publication may be reproduced by any process without written permission. Requests 
should be made to NCVER. 

The views and opinions expressed in this document are those of the author/project team and do 
not necessarily reflect the views of the Australian Government, state and territory governments 
or NCVER. 

ISBN 978 1 921809 44 6 web edition 
978 1 921809 45 3 print edition 

TD/TNC 102.27 

Published by NCVER 
ABN 87 007 967 311 

Level 11, 33 King William Street, Adelaide SA 5000 
PO Box 8288 Station Arcade, Adelaide SA 5000, Australia 

ph +61 8 8230 8400 fax +61 8 8212 3436 
email ncver@ncver.edu.au 
<http:/ /www.ncver.edu.au> 

<http:/ / www.ncver.edu.au/publications/231 0.html> 






About the research 




NCVER 



Measures of student success: can ive predict module-completion rates ? 

Jeanette Learned, Western Sydney Institute of TAFE 



Building the research capacity of the vocational education and training (VET) sector is a key 
concern for the National Centre for Vocational Education Research (NCVER). To assist with 
this objective, NCVER supports a community of practice scholarship program, whereby VET 
practitioners without research experience are given the opportunity to undertake their own 
research to address a workplace problem. Scholarship recipients are supported by a mentor, 
and NCVER publishes their research results. 

Jeanette Learned participated in the 2009 community of practice. Jeanette is a head teacher at 
Blue Mountains College, part of the Western Sydney Institute of TAFE. To help teachers 
identify when intervention strategies to prevent student drop-out are required, Jeanette 
developed a tool to predict when a student was at risk of leaving a course. 

The study involved piloting the survey tool with three classes at a medium-sized vocational 
education college in an outer suburban area. Attendance rates and final course results were 
also collected from the college records and matched to the survey data. 



Key messages 

Attendance is a key factor in students passing modules. 

The survey tool was useful in predicting whether a student was at risk of leaving. The 
overall score calculated as a result of all attitudinal variables correlated well with the final 
module-completion rate, and students with falling overall scores dropped out before the 
end of the semester. 

•y" Based on comments from some students, the author identified a potential ‘Hawthorn 
effect’, whereby the tool had the potential to improve class participation simply by 
prompting the students to reflect on their own performance. 

The finding that it is possible to obtain useful feedback about student progress in the first few 
weeks of class suggests that teachers are in a good position to influence course completions. 



Tom Karmel 

Managing Director, NCVER 
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Introduction 



A major goal of any publicly funded vocational education system is to develop a workforce with 
skills that will support the economy (Council of Australian Governments 2009). An underlying 
and reasonable assumption in achieving this goal is that students must undertake training in order 
to be skilled. 

The goal of this research project was to understand the factors that cause a student to leave a 
course part-way through delivery of the content. Most research into module-completion rates has 
been based on student experience after the event — either through interview, exit surveys or 
demographic analysis. The aim of this project was to develop a tool that could be used to identify 
factors that lead to students ‘dropping out’ — that is, leaving a course before the last class or 
assessment has been completed. 

The questions addressed were: 

Can we link students’ attitudes to study with an increased risk of leaving? 

Can we develop a tool to monitor students’ attitudes to their study? 

Can we show a link between the issues investigated and module-completion rates? 

The long-term goal of this project, beyond this exercise, is to develop a set of recommended 
procedures that would lead to better completion rates for courses. 



What is student completion? 

Studies of student completion define success in different ways. 1 It has been argued that, for 
vocational education, even partial completion of a qualification has benefits for both the students 
and the economy (Shah & Burke 2003). For this reason, where vocational education is concerned, 
module-completion rates are often used rather than graduation. Module-completion rates can be 
linked to funding (Azemikhah 2009) and are easily calculated for use by administrators as 
performance indicators. 

The module-completion rate for this study site is defined as the number of students who complete 
a unit of study, as a percentage of those eligible to complete the unit. NCVER statistics for 2008 
showed that over 77.1% of subject enrolments (Australia-wide) resulted in a subject completion 
(NCVER 2009). Module-completion rates for TAFE NSW in 2008 were 79% (New South Wales 
Department of Education and Training 2008). Completion rates for the study site, at 78.6% in 
2008, were within this range (Trevaskis 2009). 

Teachers are rarely concerned with such broad measures, but are keenly aware of the need to keep 
students participating in class. The nature of this research allowed module completion and class 
participation to be tracked, resulting in a more accurate picture of what contributes to satisfactory 
completion rates. 

Completion statistics do not reveal the stories of these students. My aim was to find out more 
about the complex set of reasons that contribute to the statistics. 



1 For a more detailed discussion on defining completion, see appendix A. 
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Literature review 



Much research has been done into factors that keep students in non-compulsory education. 

Broadly categorised, research falls into three areas. Statistical analyses of nationally collected 
samples such as student outcome surveys and Australian Vocational Education and Training 
Management Information Statistical Standard (AVETMISS) data are common, often followed up 
with interviews (either live or by phone). Some authors have used longitudinal studies (such as 
Longitudinal Surveys of Australian Youth [LSAY] data) to track student outcomes over a number 
of years. Less common are case studies and qualitative studies of small groups. There are a number 
of overseas studies that assess the effectiveness of intervention programs, but these are not always 
direcdy relevant to the Australian experience. 2 In addition, there are literature reviews that draw 
conclusions from other studies. 

Callan (2005) used AVETMISS data to identify and interview non-completing students. He 
reported that, on average, students left eight weeks after they had started and they left because of 
issues with teaching quality, course content and conflicting demands of study and employment. 
Other issues included workload, flexibility, level of difficulty, and costs. An interesting issue is that 
not all students expected to complete their study — some do not wish to be assessed, or to achieve a 
full qualification. Others have noted this, including Hauptman and Kim (2009), Hudson, Kienzl 
and Diehl (2007), Karmel and Nguyen (2007), and Shah and Burke (2003). Moreover, vocational 
education students are more likely to take longer to complete their qualification and drop in and 
out of a course. 

Other researchers have linked statistical data with demographic information to understand how 
results are affected by a student’s background (Grant 2002; Grey et al. 1999; John 2004; Marks 
2007; Shah & Burke 2003). John (2004) found that age, employment and previous education are 
factors in success — younger students, those with lower education, and unemployed students tend 
to be less successful. Marks (2007) linked longitudinal survey data to university completion data. 
The main result was that for school leavers, university entrance scores were the best predictors of 
success. Other demographic factors for university students were not critical. Uren (2001) in a case 
study of one TAFE (technical and further education) college showed that ethnic background, 
gender, employment status and type of course can be linked to different levels of success. Statistical 
analysis of this kind may help identify areas of concern. 

The factors that influence module-completion rates are not surprising. Uren’s (2001) 
comprehensive case study of student retention at a TAFE college identified choice of study and 
career plans, the teaching experience, content and workload, equipment and facilities, student 
services, finances, health, and transport difficulties as issues. Demographic factors such as age, sex, 



2 Hauptman and Kim (2009), in an international comparison of post-secondary education, suggest that the Australian 
vocational education sector is unique in terms of size (with greater student numbers than universities) and emphasis on 
‘sub-bachelor’ degrees; hence, overseas research is not always directly relevant to our VET system. In the US and the 
UK, colleges focus more on transition to university rather than job-ready qualifications (Hauptman 2008; Shreeve 
2009). The closest overseas equivalent is the US community college system. Research on completion rates for 
community colleges is often aimed at improving the participation of minority groups, rather than the completion rates 
of all students. US studies that evaluate broader campus-based programs often assume that the student is part of a 
campus-based community, as suggested by Tinto’s early research (1993). Consequently, overseas studies such as that by 
Hudson, Kienzl and Diehl (2007) do not relate easily to experiences in the Australian VET sector. 
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educational achievement and study mode (part-time or full-time) also impact on retention. 

Although these issues interact in complex ways, Uren’s report concludes by recommending 
improved recruitment and selection as important strategies in keeping students — selecting the right 
course is the first step to completing. 

Many reviewers, including Karmel and Nguyen (2007), considered the outcomes of partial 
completion, indicating that completion is not necessarily critical to a good outcome. There are 
positive reasons to leave a course; for example, getting a job, changing courses or moving into 
higher education, or because the required skills have been mastered. In some cases, students leave 
and return later to complete the same qualification, or a higher-level qualification. 

Factors that affect special groups of students such as apprentices and distance education students 
must be considered, but are not necessarily all relevant to full-time students. Kirkby (2000) 
identified job-related and personal reasons as the most common, but also confidence and 
motivation (20—25%) and teaching-related issues, including course content (19%), delivery (11%), 
teachers or assignments (10%) as factors causing students to leave. 

Many authors have reviewed the success of apprenticeship programs (Snell & Hart 2007a, 2007b, 
2008; Mahoney 2009). Completion rates are as low as 50%, but most reviews have linked job- 
related issues rather than the formal training component to the attrition rate. Students reported 
work-based factors such as poor pay, work conditions and lack of suitable workplace training as 
more critical to non-completion than TAFE-based training. In fact, Snell and Hart (2008, pp.58— 9) 
found that students generally were positive about training when it was TAFE-based. However, 
some reasons for non-completion common to general students do emerge; for example, availability 
of support for class work and the age and level of education of the candidate. 

So what can be summarised about student retention? Vocational education students in Australia are 
a diverse group in terms of age, goals, employment status and educational background, but are 
more likely to value specific skills rather than a pathway to higher education or even a completed 
qualification. Statistics show that outcomes can be linked to a wide range of demographic factors 
and that students leave for a predictable range of reasons, including course difficulties, demands on 
time and changes in plans. 

Apart from surveys of students identified by statistical data and analysis of exit surveys of 
completing students, there is a lack of student-centred data, and there is no information on what 
happens before a student leaves a course. Most surveys are conducted after the event. This research 
was an opportunity to obtain first-hand information on student engagement with course delivery 
and to link those data to student outcomes. 

Improving completion rates 

A number of detailed reports between 2000 and 2004 identified factors that prompt students to 
leave a course and suggested possible solutions. Relevant issues from the work of Polesel, Davies 
and Teese (2004), Uren (2001), Martinez (2001) and Mclnnis et al. (2000) are discussed below. 

Uren (2001) interviewed students with low module-completion rates to determine problem areas 
that could be addressed. She concluded that ‘individual motivation or commitment’ was the 
primary influence on students withdrawing, but suggests that there are a number of strategies that 
can improve outcomes. These include improved pre-enrolment information; selection interviews; 
better orientation; better provision of counselling; transition programs for school leavers; and 
procedures to follow up and assist students who miss class. 

In a detailed literature review, Mclnnis et al. (2000) note a lack of research to back up theoretical 
work and a shift in focus from student and sociological issues to the role of the institution as 
reasons for student withdrawal. Although in Australia there have been changes since 2000, areas of 
concern remain remarkably constant. Most of the concerns mentioned in this report continue to be 
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discussed. While Mclnnis et al. (2000) and Uren (2001) both note that students leave for a complex 
set of reasons, the issues most relevant to VET students can be categorised as relating to: 

V Choice of course-, field of study, relevance of content, commitment to the course, expectations and 
what actually happens. 

Quality of experience-, quality of teaching, workload, difficulty of the work, study skills, support 
available, progress made. 

V Personal-, often considered as beyond the control of the institution, Mclnnis et al. (2000) note 
that the reality is more complex; for example, student motivation and study difficulties are 
dependent on classroom instruction. 

V Financial-, usually referring to conflict between study and the need to work — issues which 
become more critical as responsibilities increased. 

Mclnnis et al. (2000) suggest improving the recruitment process, managing the transition to study 
and improving student support initiatives; for example, identifying ‘at risk’ students and providing 
early intervention. The issue that many students leave without consulting anyone is of concern and 
is raised by other authors (Uren 2001). 

Grant (2002) investigates factors that prevent students from completing a course. He identifies 
factors that lead to partial completion (rather than full completion). These include change of job, 
type of course (course length and qualification level) and organisational factors. Despite noting a 
high module-pass rate (over 50% of students pass all the modules they are enrolled in), and making 
no judgment on the value of partial completion (p.14), his results suggest that students would 
prefer to complete their qualification, but that various issues have intervened. 

While it has been argued that partial completion is acceptable as not all students intend to complete 
their qualification (Karmel & Nguyen 2007), Polesel, Davies and Teese (2004) note that most 
(approximately 90%) of the students surveyed in their study said that they intended to complete 
their qualification. 3 They also comment on the diverse nature of VET students in terms of reasons 
for studying, educational background and the time they allow for completing a qualification. 
Students surveyed were very positive about training but identified a lack of support, such as 
information on careers and counselling. Statistics link many factors with a tendency to ‘drop out’, 
including lower school achievement, struggling in the initial stages of a course (40% of whom 
dropped out), younger students and especially males. A critical point is made that surveys are 
answered by successful students, rather than those who leave. 

Callan (2005) addressed this issue by surveying a sample of the 1 6% of students who leave TAFE 
with no record of achievement. Student backgrounds were characterised by: 

% educational background — one-third had only completed up to Year 10 

V age — the modal age was 1 8 

-v* employment — students were more likely to be unemployed 
■y- reason for doing the course was interest (40%). 

The reasons given for leaving include those already mentioned, but with more emphasis on 
teaching quality, content (difficulty, not matching needs) and course (flexibility and workload). 
Callan makes some additional points: 

% Most of the students said that they would return to study if teaching quality, course content and 
flexibility of hours were changed. 



3 Intention to complete is a difficult measure to capture reliably as it can be influenced by many factors including how a 
student defines completion (see Polesel, Davies & Teese 2004, p.66 — ' 7) and a social desirability bias, where students feel 
obliged to say that they will complete. In our survey students were asked whether they think they will complete on a 
number of occasions. Responses varied greatly over the semester and correlated with measures such as attendance, 
work done, feeling comfortable in class and getting enough help. 
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•y" It is likely that students are unaware of their options for study and career in a ‘large and diverse 
system’. 

Many of these students actually completed the course the following year. 

Students who leave are often more critical of the institution than successful students: 

There is a well-established finding in educational research that students attribute failure to 
external factors (for example, the quality of the teachers and course content) more than to 
personal factors (for example, their lack of effort or motivation). (Callan 2005, p.23) 

Some interesting issues were raised when teachers were surveyed about the students who had left. 
Teacher comments included: 

Some students had litde outside support and were experiencing financial problems. 

The course selection process allowed students into a course who were ‘doomed to fail’ because 
of poor educational background and poor understanding of the level of content. 

Better support for such students should include night classes, more flexible transfer and 
recognition arrangements. 

Segmentation may help with such students; that is, recognising the needs of different student 
groups and organising support services, especially in the first few weeks of a course. 

John (2004) identifies key factors that can be linked with a lower chance of passing. These include 
those already mentioned (younger students; unemployed; those with lower educational 
backgrounds), as well as equity group factors (gender, age, ethnic background, first language). The 
concern of this report is that in striving for better performance we may eliminate those students 
who most need the system. 

The most recent reports follow up somewhat on issues already mentioned, suggesting that this is a 
perennial problem which is still in need of solutions. Misko and Priest (2009) analyse NCVER’s 
Student Outcomes Survey of 2006 to categorise suggestions for improvement. Misko and Priest’s 
work reinforces suggestions such as the importance of teaching that is relevant, current, directed 
and efficient, as well as the need for useful feedback on performance. Other interesting comments 
were that there should be mandatory attendance; that students at times disadvantaged others by 
being late; and that there should be graded assessment. 

The literature review raised the following issues. First, much of the research is based on analysis of 
statistical data. There is a lack of information gleaned from more detailed, student-centred local 
studies. Second, much of the information has been gathered, with the exception of Callan (2005), 
from successful students. There is a need for more information from students with a less-successful 
student experience. Third, most studies are based on research data obtained after the event — not 
while students were still attending class. 

This study was able to track some of the issues raised as reasons for leaving from students while 
they were attending class. Issues investigated were: 

•y" satisfaction with teaching (help, understanding and how well they ‘fitted in’ with the class) 
how other life issues affected study 
■y" if students expected to finish the course 

if they were satisfied with their own class participation (work and attendance) 

•y" how career information and course selection affected retention. 

If these issues were factors in students leaving, then being able to identify them along the way 
would be the first step to providing interventions that worked. 
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Background information 



The study site was a medium-sized vocational college in an outer suburban area in New South 
Wales. The survey was run in three sections of the college, in classes at certificate III level. One 
class was general and the other two had a clear vocational outcome. Classes comprised students 
from mainly English-speaking backgrounds of varied age and educational background. 

To understand the link between module completion and class participation, some preliminary data 
analysis was done on available statistics — at a national level and for the study site. 

Australia-wide results for subject enrolments and results are shown below. This table is derived 
from the NCVER students and courses report for 2008. 



Table 1 Subject results as a percentage of subject enrolments, 2004-08, Australia 



Subject result 

(% of total enrolments) 


2004 


2005 


2006 


2007 


2008 


Assessed — passed 


69.1 


69.4 


68.4 


68.3 


68.1 


Recognition of prior learning 


2.6 


2.7 


3.2 


3.3 


4.1 


Recognition of current competency 


- 


- 


- 


0.1 


0.1 


Recognition of prior learning/ 
current competency — not granted 






" 


0.1 


0.1 


Assessed — failed 


7.5 


6.6 


6.4 


6.5 


6.6 


Withdrawn 


8.9 


9.3 


9.4 


9.1 


8.8 


Continuing studies 


6.5 


6.2 


6.7 


7.3 


7.1 


Not assessed — completed 


5.1 


5.4 


5.5 


5.1 


4.9 


Not assessed — not completed 


0.3 


0.3 


0.5 


0.3 


0.3 


Total subject enrolments 


100.0 


100.0 


100.0 


100.0 


100.0 



Notes: Percentages calculated from the original table. Due to rounding figures may not equal 1 00. 
Source: NCVER (2009, p.17). 



Module-completion rates are calculated, based on the number of pass results relative to the number 
of pass, fail and withdrawn results. In the table above, the withdrawn rate is higher than the fail rate. 

In my experience, withdrawn results are most often given when a student is no longer in class when 
assessments are finalised. The definition of withdrawn is that: 

This code is recorded where a student has commenced but not completed a module/ unit and 
there is evidence of their participation, such as a record of class attendance, work submitted, a 
result for an assessment or communication between the teacher and student, showing the 
student has commenced study in the module/unit. (TAFE NSW Western Institute 2010, np) 

Fail is defined as: 

. . . used when a student has commenced, but not successfully completed a unit or module. 

Issued when there is progressive assessment at a failing level. 

(TAFE NSW Western Institute 2010, np) 

Anecdotal evidence from teachers suggests that students who ‘disappear’ during a semester are 
most often awarded a withdrawn result for all units they have started, but not completed. Many of 



12 



Measures of student success: can we predict module-completion rates? 



the researchers have noted that students typically drop out of a class halfway through the 
semester — at about week eight. At this point, they have attempted most of the units in a class 
grouping, but will usually not have completed assessments for any unit. 

To investigate this idea further, records from the student enrolment system at the study site were 
used. The results below are collated from student results for the study site for 2009. Results were 
calculated against attendance levels based on hours attended as a percentage of total possible 
attendance. 



Figure 1 Module-completion rates by attendance level 



Attendance range (%) 




Source: Student enrolment data, study site. 



This chart shows that most pass results are gained by students with a recorded attendance level of 
over 80%. The highest numbers of withdrawn results are awarded to students with attendance 
levels below 60%. The highest number of fails were awarded to students with attendance rates 
below 20%. There is an overall trend for lower fail/ withdrawn rates as attendance increases, with 
only 5% of students with attendance of over 60% receiving a fail or withdrawn result. In fact, the 
module-completion rate for students in the highest attendance bracket was 91%. 

Similar results were obtained for other sections, other colleges and other years. 

These data are hardly surprising. Students who attend most of the classes are usually not recorded 
as withdrawn and have a high chance of passing. In this and other results, results recorded as fail 
for students with an attendance of over 80% made up less than 3% of the results. 

When the module-completion rates are reviewed again in the light of these data, it is apparent that 
more than half of the modules recorded as not complete are the result of students who have 
either left the course halfway through, or have such low attendance rates that they are not able to 
be assessed. 

Reports from students who leave courses (Uren 2001) suggest that missing class is a cumulative 
problem — the more classes missed, the easier it is to drop out. Other students state they ‘just stop 
going’ to class (Uren 2001, p.10). The high pass rate for students with high attendance suggests that 
failure is not as much a problem as failing to participate in learning. Uren’s report supports my 
observation that leaving a course is often simply an accumulation of missed classes. 
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Comments from teachers support the importance of attendance: if a student is not in class and is 
not participating, it doesn’t matter how good the teaching or facilities are; they are not gaining any 
benefit from being enrolled and will probably not pass. Of course, there are many reasons for not 
attending class. Students typically have appointments, are not feeling well, are late or have critical 
issues to attend to. It is likely that other factors are involved and these reasons just make it easier to 
miss a class. It was hoped that the survey questions would uncover some of the real reasons. 

Low attendance is not only an issue for the student. Teachers often design courses around projects 
and team work, to simulate workplace environments. Such projects are difficult to manage when 
students do not attend, and unreliable students often end up being left out. There are other issues 
such as morale problems (for other students and the teacher) and continuity. It can be frustrating 
for other students when time is spent ‘catching up’ students who have missed work. At a section 
level, dwindling class groups become a drain on resources, especially where a course covers a 
number of semesters. 

This background information does not solve the problem of low module completion; however, it 
did focus my research on finding out why students stop attending. 



A note about course delivery 

There are many ways of delivering course content, and there has been much encouragement in 
recent years for more flexibility and innovation in delivery. Previous research has noted that 
students who leave would like courses to be run at times other than weekdays, such as at night or 
weekends (Misko & Priest 2009; Callan 2003). Statistics show that the majority of TAFE students 
are part-time. In 2008, 87.5% of students in the public VET system were studying part-time 
(NCVER 2009, p.9). 

College experience at the study site was somewhat different. Although classes are offered at night 
and by flexible arrangements, most students elect to attend classes during the day. Students who 
have been offered flexible arrangements often re-enrol in a formal class. Part-time attendance for 
colleges with smaller class sizes often means a timetable made up of a subset of the classes taken 
by full-time students. The experiences of part-time students in these classes are similar to those of 
full-time students. 

Flexible delivery options have many issues that need further investigation; for example, Grant 
(2002) notes that courses with less choice have higher completion rates. Kirkby (2002) notes that 
attrition rates are higher for externally delivered courses than for ‘face-to-face education providers’ 
(p.l). These issues are not addressed in this study. This research project was specifically directed at 
students who have elected to attend classes (either as a part-time or full-time program) during the 
day, for the reason that this included the majority of the student population at the study site. 
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Research methods 



Information on student completion is typically gathered by exit survey, enrolment statistics or 
surveys of students who have either left or completed a course. The aim of this project was to 
develop a tool that could be used with students while they were still studying. The assumption was 
that an effective tool would be a predictor of student completion, but more importandy it could 
detect information about a student’s progress and factors that might lead to the student leaving the 
course. A related outcome might be information to help develop targeted intervention strategies for 
those students identified as at risk of non-completion. 

Most teachers are able to identify factors that cause students to leave, and there are many informal 
intervention strategies used by teachers. The original plan was to develop a more formal strategy, 
with a survey that could be answered quickly on a regular basis by students in the first few weeks of 
class. Results from this survey could then be linked to outcomes obtained from class attendance 
records, final results and interview. Ideally, surveys would be computer-based to enable data to be 
processed efficiently. 

A pilot study was set up for second semester 2009, with three classes at the study site (44 students). 
The classes were chosen on the basis of convenience, as the first information garnered was to be 
used as a pilot study. All classes were at certificate III level and in three different teaching sections. 

The survey tool comprised two questionnaires. In the first few weeks of attendance an initial 
questionnaire was used to collect demographic information and background information relating to 
how the student had selected their course and the career advice they had received. The second 
questionnaire was aimed at identifying students at risk of leaving. Students were asked to complete 
this questionnaire once every two weeks for up to 12 weeks. Questions were chosen to reflect 
factors identified in previous studies as reasons for leaving. The survey questions are included in 
appendix B. 

The structure of the questionnaire was a series of linear (visual analogue) scales, with low values to 
the left and high values to the right. Students were asked to rate their answers by placing a mark on 
the line between two extremes. The mark was converted into a number between 0 and 10. A 
numeric value was not attached to the scale to prevent students comparing results (with themselves 
or others) and to encourage a more intuitive result. 

The initial survey was electronic, but it quickly became apparent that a paper-based survey was 
quicker, easier for the students, and more effective. In order to encourage students to be honest in 
their answers, information was not collected by teachers direcdy. To make the survey quick and 
practical, all of the questions appeared on one page. 

Not all students answered every survey, due to absence or choice. Datasets with only one response 
were excluded from the data analysis. 

Students in the surveyed classes were followed up with interviews, both in class and individually, 
where the survey results were of interest. Information on the success of individual students was 
obtained from enrolment records. Although individual students were tracked, they are identified in 
this report only by an arbitrary identification number. 
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Practical considerations 



Students were generally pleased to complete their survey each week. Most teachers were supportive 
of the survey, and the response rate was good, with only one or two students per class not agreeing 
to take part. Feedback during the weeks being surveyed included: 

a request to have a space for comments on the form 

refusal to complete the survey because the student ‘did not intend to drop out’ 
a few complaints in later weeks — ‘not this again’ 

•y" students very keen to know ‘how they were going’ — they were very interested in the results. 

There were some problems with the delivery and administration of the survey. Many TAFE courses 
include communications classes, which often incorporate activities designed to reflect on learning. 
Originally, the intent was to include the survey in the content of these classes. This conflicted with 
the desire to maintain student confidentiality, with class teachers not seeing the results. In the pilot 
study, however, student feedback indicated that, where a class group had a number of teachers, this 
was actually not a concern. The second issue was that in a busy teaching schedule, surveys were 
forgotten. This in some cases resulted in surveys not being completed by students who were 
missing classes — and these were the students we needed most data from. 

Another issue is that surveys, to be useful, must be handed out in the first week of attendance. 
Ideally, the career questions would be completed at enrolment or orientation. To achieve this, the 
purposes of the survey need to be conveyed to teachers, and teachers need to support the process. 
The tool is only useful if it is perceived as returning useful information. 

There was a suggestion that, in the process of reflecting on their own progress, students would 
actually make better progress, meaning that the results of the survey would not be valid — the well- 
known ‘Hawthorne’ effect, as described in Elton Mayo’s experiments of 1927— 32. 4 This was one of 
the possible results of the trial, but was not seen as a problem. In fact, if there was an improvement 
in student completion, the survey would become part of the solution, rather than just a tool to 
diagnose the problem. 



4 The major finding of Mayo’s study was that almost any change in the working environment improved production. The 
conclusion was that the workers responded to the attention from researchers by working harder. 
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Findings 



The surveys resulted in a matrix of statistical data. For each student, there was a set of basic 
demographic data (age, gender, language and educational background) and answers to two 
questions aimed at finding out how the course linked to student career plans. In addition, 
information on class participation was gathered over three to four months. 

Class participation was determined by monitoring classes in the last two weeks of semester. 
Additional data on attendance and module completion were extracted from enrolment records after 
the semester had ended and individual results were matched with the official statistics. Attendance 
records were used to get a more valid measure of participation; for example, a student showing not 
present in the last week may have had perfect attendance but have finished early. Likewise, some 
students attend in the last week, despite having missed significant numbers of classes. 

At the end of the semester, a class meeting was held to get feedback on the process, and to discuss 
the results. The main questions addressed at the meeting are included in appendix B. 



Initial survey (demographic and career data) 

Because of the small sample size, demographic data were not used. There are indications in the 
research literature that factors such as age, previous education and employment have strong links to 
student success. The student groups surveyed included individuals in most of these categories, but 
not in sufficient numbers to justify data analysis. 

The two career-related questions were included to identify more information on the link between 
module completion and the reasons for doing the course. The first question is labelled below as 
‘career goal’, the second as ‘course reason’. 5 



Figure 2 Career question — initial student survey 

I am certain of the career or 
job I want to follow 



I have no idea what career 
or iob I want to follow 



I am doing this course 
mostly for personal interest 



click on the bar to respond 



I think this course will help 
me achieve my career goal 



In each class group there was a wide range of responses. When sorted by the value for career goal 
(where 0 indicated no idea of career choice and 10 indicated certainty), it appeared that students 
who were sure of their career goals were also more likely to give a high rating for course reason, 
indicating that they had a career goal and that the course was linked to this goal. Most of these 
students had good attendance and module completion (only one student, K76, had a module 
completion or attendance rate of less than 75%). This was expected, and supports the literature that 
suggests that good career advice and course ‘fit’ is an important factor in student retention. 



5 In retrospect, a scale which rated ‘personal reasons’ against ‘career goal’ may have confused two issues, and in 
subsequent surveys this will be broken into two questions. 
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It is interesting that students who were not so sure about their goals could be quite positive that 
the course was going to help them with their career (L75, K35, J53 and S51). Surprisingly, all but 
one student (S51) in this category completed most modules, despite their attendance being lower 
than 65%. 

Student S77 was the only student to give a ‘middle’ score (between 4 and 6) for career goal and a 
very high score (9.8) for being sure that the course would help his/her career. This student did very 
well, completing all modules and attending 90% of classes. All but two students (J 00, R23) who 
scored in the middle range for career goal also scored above 4 for course reason. In this group, 
there were no apparent links to outcomes from this set of data. 

The profiles of the eight students mentioned above are shown below, and are worth some 
consideration. 

As shown in table 2, student S51 was young (16—17), had completed only to Year 9 with no other 
education, had very low attendance (15%) and only completed three modules. The other three 
students in this subgroup were either older or had a better educational background. Student L75 in 
fact had health issues, and had completed all modules by the beginning of 2010. 



Table 2 Student profile — low career goal or course reason 







Student profile 




Course performance 


Student 


Age 


Highest 


Post-school 


Module- 


Attendance 




range 


school year 


education 


completion rate 




No strong career goal, but course reason as ‘help with career’ 






J53* 


16-17 


Year 1 1 


Certificate III 


93% 


50% 


K35* 


40-44 


Year 10 


None 


76% 


65% 


L75* 


20-24 


Year 12 


Certificate II 


58% 


48% 


S51 * 


16-17 


Year 10 


None 


33% 


15% 


Course reason primarily ‘personal reasons’ 








J00** 


18-19 


Year 10 


Certificate 1 or 
Statement of 
attainment 


71% 


43% 


S77** 


16-17 


Year 10 


Certificate II 


100% 


90% 


P83** 


20-24 


Year 12 


None 


47% 


55% 


R23**‘ 


40-44 


Year 12 


Advanced 

diploma 


100% 


82% 



Notes: * Students with scores indicating ‘course will help with career' but 'no career goal’. 



** Students indicating personal interest as the reason for doing the course. 
Source: Survey data collected from students. 



J00, despite having low attendance, managed to complete 71% of the modules in which he/ she was 
enrolled. It is interesting that a young student with no strong career goal and little post-school 
education indicated personal reasons for doing the course, but this may be an indication that the 
student was still deciding on a career path — and perhaps better career advice would help. Students 
S77 and R23 are discussed above; P83 is mentioned later. 
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Figure 3 Course-career link — ratings for career goal and course reason (sorted by career goal) 




This example highlights both the potential of the survey and the need for a larger sample. Students 
with profiles similar to S51 (young, low school achievement, no post-school education) have been 
identified in many reports as ‘high risk’ for non-completion (Callan 2004; Grant 2002). This student 
and K35 were in the same class, yet the older student succeeded. 

The course reason question is really a combination of two measures — career purpose and personal 
interest. Only two students gave a very low score for this question, indicating personal interest 
rather than career. The outcomes were quite different. R23 was older, with good previous 
qualifications and had passed all units with a high attendance rate. P83 should have been a good 
candidate, with a good school background and 20—24 age group, but perhaps personal interest did 
not generate enough motivation to complete. 

There was a slight correlation between module-completion rate and both of these variables (career 
goal and course reason) for this sample. The correlation increased when both measures were 
combined, indicating a strong career focus and possibly therefore a strong motivation for 
succeeding. Correlation values increased for the classes with a clear vocational outcome, but almost 
disappeared for the general class. Students start courses for a complex set of reasons which are not 
always career-related, but the patterns noted confirm results in the literature that suggest that good 
career advice can help improve student outcomes. 

The individual case studies indicate also that these two career-related measures may be significant 
when considered with other background information, such as age and previous education. 

Table 3 Correlation between career scores, module-completion rates and attendance 

Correlation values — career information 





Combined career 
score 


Number of pass 
results 


Final module- 
completion rate 


Number of pass results 


-0.11 


- 


- 


Final module-completion rate 


0.29** 


0.66*** 


- 


Days attended (%) 


0.24** 


0.48*** 


0.79*** 



Notes: * significant at p = 0. 1 ; ** significant at p = 0.01 ; *** significant at p = 0.001 . Values over 0.5 are considered ‘strong’ 
correlations. 

Source: Calculated from student survey data (all student results). Module completion and attendance rates extracted from 
student enrolment records. 



Student feedback 

Comments about the survey process from students at the end of the semester were generally 
positive and included: 

a general agreement that the survey showed that teachers were more interested in student success 
students keen to know the results of the survey. 

Students were interviewed in a class meeting and responded to the following questions, the answers 
to which are summarised below. 



Did you find any of die survey questions difficult to answer? 

In general, students considered that the questions were easy to understand and answer. A minority 
of students thought that a numeric scale would make their answers more accurate — partly because 
they wanted to use the result as a comparison. 

The question about life being busy was discussed. Students were busy in general, but they did not 
see life’s busyness as being connected to their TAFE studies. A suggested rephrasing of the 
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question was ‘is the TAFE workload making life difficult’. Some students would have preferred to 
indicate which teachers were not giving them enough help. 

Were you able to give accurate responses? 

Students would have liked a comment space to write more information. When this was provided, 
some interesting insights were gained (comments are reproduced in the appendix C). 

Because some classes were good and others not good, it was difficult to make an overall judgment. 
Responses may differ according to the class in which the survey was distributed — classes where 
students like the teacher would give better responses. In general, students felt that they could 
answer accurately because teachers were not identified. 

Did answering the survey have any effect on your involvement with the course? 

Student S51 said that she was determined to finish the course, because of the survey. Another said 
that answering the questions made her think about her performance. Others agreed that being 
asked each week about their progression made them think that the faculty cared about them, that 
they had taken the time to ask and that their voice was being heard. 



Were there any questions that were ambiguous; that is, not clear in their meaning? 

There was a unanimous ‘no’ to this question. 



What happened in class that caused a ‘dip’ in all responses in week 6? 

This was specific to one class but raised some interesting issues. Students commented that they 
were overloaded with assignments at about this time in the semester; that a teacher was away in 
week 6 and extra work was given the week before; they were working at home to be more 
productive; there was a lot of information to be understood at one time; and there was time wasted 
in class because of a technical issue (software not working properly). 

The importance of these issues cannot be underestimated. If the whole class experiences difficulty, 
then students who are struggling may well drop out. Organisational factors such as continuity 
between class teachers and replacement teachers and good facilities may be critical. 

Was there any other information that you would have liked to include? 

Students suggested asking: 

if people were attending just because of Centrelink benefits 
if you have the opportunity to be elsewhere, where would that be 
•y" more questions on career goals. 

Students also said that they got to a critical point in class where they had to decide if they wanted to 
continue or not. Better career information at this point would help. Over half the class said that 
they would like more career information. 



Survey responses (course progression) 

One-hundred-and-thirty-one responses were collected and analysed for both correlation between 
variables, and patterns in individual responses. Responses were also matched with student 
outcomes for attendance and module completion. To simplify analysis, the questions were 
categorised as: 

■y" Like score (I liked nothing/ everything in the course) 
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•y" Understand score (I understood nothing/ everything in the course) 

■y- Attend score (I have attended none/all of my classes) 

•y" Work score (I have done no work/ sufficient work on assignments) 

Life score (Life is too busy/ easy) 

•y" Help score (I am not getting any help/getting enough help from my teachers) 

■y* Complete score (I do not think I will complete/ am sure I will complete) 

Class score (I am not at all/ completely comfortable in my classes) 

•y" Overall score (an average of the other eight responses). 

The most striking result was that the drop-out rate for one class (the business class) was almost nil. 
Information from students was that the survey helped them reflect on their own performance and 
made them feel that ‘someone cared’. These effects were not as striking in the other two classes, 
possibly because there was not as much involvement in the project and much less feedback was 
given to students. 6 

When student responses were tabulated, there was a reasonable correlation between some of the 
measures and the actual student results. Table 4 shows correlation values between the last survey 
result and student outcomes. As there were many survey responses for one student outcome, to 
avoid weighting those student responses where there was more than one, a single value was chosen. 
Results were similar even if the highest score or the average score was used. 



Table 4 Correlation between survey measures, module-completion rates and attendance 





Like 

score 


Understand 

score 


Attend 

score 


Work 

score 


Life 

score 


Help 

score 


Complete 

score 


Class 

score 


Overall 

score 


Module- 
completion rate 


0.34*** 


0.33*** 


0.44*** 


0.45*** 


0.37*** 


0.12 


0.12 


0.15 


0.48*** 


Number of 
passes 


0.09 


0.15 


0.28** 


0.36*** 


0.25** 


0.05 


0.17* 


0.03 


0.32*** 


Attendance (% 
of classes) 


0.35*** 


0.35*** 


0.39*** 


0.46*** 


0.39*** 


0.11 


0.11 


0.07 


0.43*** 



Notes: * significant at p = 0.1 ; ** significant at p = 0.01 ; *** significant at p = 0.001. 

Source: Survey measures taken from student survey data, module-completion rates and attendance from enrolment records. 



The most notable correlation was between the overall score and actual results for module- 
completion and attendance, suggesting that survey results could be used to predict success. Scores 
for attendance, work and also liking, understanding and life were also linked but to a lesser degree. 

The low (or zero) correlations are interesting. Liking, help and class scores showed a low 
correlation with the number of passes. A student can have a high module-completion rate if they 
passed only one unit, but the number of passes is an indication of how many units were completed 
over the semester and thus how much of the course had been completed. Students who elect to 
attend part-time and who are doing well can have a high module-completion rate, but a lower 
number of passes. 

Scores for help, expecting to complete and being comfortable in class did not correlate with actual 
attendance, but the amount of work done, understanding and liking did. This supports anecdotal 



6 Although the survey was run as described for each of the three classes, I administered the survey questions for this 
group, and as a result more feedback on the purpose and progress of the research was given (informally) to these 
students. The other two classes completed the surveys electronically or with another teacher, and did not have as much 
opportunity to ask questions. 
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evidence (and also Uren 2001) that non-attendance is a compounding factor: as more work is 
missed, less is understood and students begin to dislike the course. 

There was some indication that students have an unrealistic idea of how often they have attended 
class. The student measure of attendance and the actual attendance result showed a slight negative 
correlation for students with low module-completion rates (less than 50%). Although the two 
measures were taken at different times, this does indicate that the student’s own estimate of 
attendance did not predict the final attendance. It is possible that students did not realise until it 
was too late that missing class has impacted on their ability to complete the course. Comments 
recorded on the surveys support this theory, as does classroom experience. 

There was also a low correlation between the expectation to complete and the module-completion 
rate. Moreover, students with a module-completion rate of less than 50% were quite likely to score 
8 or 9 for completion. Once again, there appears, at least for some students, to be an unrealistic 
expectation of what is required to complete. 

There were also some interesting correlations between scores, some of which help explain what 
contributes to a higher overall score (appendix D shows all correlations): 

•y" help and understanding — where a student gets enough help, they have a higher score for 

understanding. If they don’t get enough help, their understanding score is lower. ‘Enough help’ 
will be different for each student 

A understand and complete; help and complete — where a student understands, they feel they will 
complete. More importantly, not understanding may contribute to the decision to leave 

attendance and work completed — this is interesting, especially given one of the class comments 
that they stayed home to get work done. This link may support teacher observations that 
students are often not as motivated to do work at home, and often find it difficult to catch up 
even when materials are made available online or as handouts 

most of the other scores showed some degree of correlation. With the sample size, values over 
0.305 are significant at the 0.1% level (p = 0.001). With more data, these correlations may be 
further explored. 

Again, the unrelated scores are interesting. Lilting a course and attending showed no correlation. One 
of the highest-achieving students in the business course (N51) consistently recorded that she ‘hated 
the course’, yet her module-completion rate (100%) and attendance (94%) were amongst the highest 
(as calculated from actual enrolment records). 

When this student’s progress was charted, it was apparent that all other scores were high or 
improving — the other measures made up for one or two difficulties. This can be seen in figure 4. 
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Figure 4 Student progress chart (high module-completion rate) — student N51 




Notes: Score for like' not charted, as the value was consistently below 0.1 . Overall score is an average of all other scores. 



Students with a low module-completion rate generally showed successive ‘falling’ scores on more 
than one measure (figure 5, student with module-completion rate of 30%). This was more apparent 
in the overall score, which was an average of all other scores. 



Figure 5 Student progress chart (low module-completion rate) — student P85 




Individual survey responses (tracking student progress) 

Individual responses were difficult to generalise if each score was considered separately, because of 
the number of variables used. However, as shown above (figures 4 and 5), the ‘overall’ score (that 
is, the average of all scores for a particular date) was a reasonable predictor of success. Students 
with module-completion rates of over 80% generally showed a score of over 8 for most measures. 
The exception was ‘life’, where even students with 100% module-completion rate showed low 
results at times. Table 5 shows the average of scores for students. When students were grouped by 
attendance (at 75%), the average was lower for all measures for the group with lower attendance. 
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Table 5 Average scores and attendance 





Like 

score 


Understand 

score 


Attend 

score 


Work 

score 


Life 

score 


Help 

score 


Complete 

score 


Class 

score 


Overall 

score 


Attendance 

<75% 


6.38 


6.65 


7.36 


6.01 


3.59 


7.80 


7.34 


7.27 


6.55 


Attendance 

>75% 


7.43 


8.19 


9.36 


7.96 


6.77 


8.99 


9.01 


8.55 


9.47 



Notes: Values are calculated from student responses. Range of possible values is 0-1 0. 
Source: Student survey data for students grouped by attendance level. 



This fits with student feedback that they were busy, but their course was just part of that busyness. 
However, it does not fit with the very common excuse that students give for not completing work 
or attending class, which is often that life got in the way (one way or another). The much lower 
average for ‘life’ for the low attendance group may be an indication that this is a contributing 
factor. As indicated by the student comments, participation is affected by issues unrelated to study 
and tends to compound other factors. 

The most useful information was obtained by graphing the ‘overall’ score; that is, the average of all 
scores given by one student on a particular date. When overall scores for an individual student were 
graphed, the results were predictive of poor outcomes. Successive ‘falling’ scores, as shown for P85 
and P83 in figure 6 were linked to low module-completion rates and poor attendance. Students 
with high module-completion rates and attendance (such as N51, D38) did not score more than 
one ‘falling’ score in a row, as shown in figure 7. 



Figure 6 Overall survey scores — low module-completion rates 




Notes: Overall score is the average of all measures for a particular survey date. 

Average score is the average of all students overall scored for a particular date. 
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Figure 7 Overall survey scores — high module-completion rates 




Notes: Overall score is the average of all measures for a particular survey date. 

Average score is the average of all students overall scored for a particular date. 



When the overall scores for an individual were graphed against the average (of overall scores) for all 
students, the effect was more obvious and corrected for events which affected the whole class. As 
noted in the interview with students, there were times when all students in a class gave low scores 
for some measures. By comparing individual results to class results, these differences are accounted 
for. For example, figure 6 shows the scores for two students against the average overall scores for 
the whole group. A more realistic result was obtained by comparing individual scores to percentile 
ranges for a class group. Students with poor outcomes (low attendance and/ or low module- 
completion rate) always scored in the lower quartile of scores for their group. 



Predicting success (or risk of failure) 

In conclusion, it appears that the best prediction of student success was obtained by comparing a 
student’s overall score to the overall score for the class. Students whose scores fell in the lower 
25th percentile were at risk of not completing (75% did not complete the semester). Students with 
two consecutive ‘falling’ scores (that is, where the overall score was lower than the previous overall 
score) did not complete the semester. This suggests that students’ attitudes towards study (as 
captured by the survey), and changes in these attitudes, can indicate whether or not a student is 
likely to succeed. 



Student comments 

After the first week, surveys included a space for comments. Student comments are reproduced in 
appendix C, and are summarised below. 

Over half of the comments were explanations of why students felt they were falling behind. 

Reasons given included illness, work and personal issues. Students were aware that they were falling 
behind, but did not ask for help or include their strategies for catching up. 

There were a number of positive comments suggesting that students found the course challenging, 
the teachers were good and classes were flexible. This supports research based on surveys of 
successful students who give positive feedback about their TAFE experience. 
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The most concerning comments were indications of frustration and helplessness. One student said 
that his teacher was away and he was dropping the class; another that he’d missed a week and was 
overwhelmed with work. 

It is interesting that students felt a need to explain their problems. In some cases, the issues 
mentioned could have been easily addressed if there had been someone to listen. In this context, 
the survey tool could be a means for students to seek help. There has to be effective help available. 
Most colleges have a counselling service and provide learning support for all students. 
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Conclusions 



Module-completion rates were shown to be highest when students had good attendance. For the 
study site, the greatest contribution to low module completion was from students who received 
‘withdrawn’ results for units. In the study group, no student who received multiple withdrawn 
results had recorded attendance of over 50%. Only two students with reasonable attendance (over 
50%) failed a unit. In most cases, a withdrawn result was given because the student was not in class 
for assessment. 

While there are other ways for students to learn, this study focused on class-based enrolments. For 
these classes, low attendance causes problems with continuity of understanding, group work and 
general class morale. Student feedback strongly indicated that when classes were missed, the 
workload became more difficult — in some cases, overwhelming. Other feedback indicated that the 
support of other students and teachers was important when students did miss classes. Support in 
the form of notes for classes missed, and just encouragement that they could catch up on work they 
had missed, could be helpful. 

The survey tool was useful in predicting that a student was at risk of leaving. The overall score, 
talcing into account all variables, correlated well with final module-completion rates. When 
individual results were charted against the class scores, 75% of students with an overall score in the 
lowest quartile dropped out. Moreover, students with falling overall scores dropped out. 

The initial survey had the potential to identify students who were in ‘high risk’ categories. Although 
the sample size was small and not statistically significant, results were consistent with other studies 
that showed that students who are younger, who have poor career goals, low educational rates and 
are unemployed have more trouble completing their course of study. These initial results indicate 
that a survey of this kind could be used at or before enrolment to identify high-risk students. 

The variables used in the survey tool highlighted some of the factors influencing student 
participation in class. A critical result was the difference in results for all students, indicating that 
success is a combination of different factors. Some correlations did stand out; for example, 
understanding and enough help were strongly linked. With a larger sample size, it is possible that 
student background could be linked to responses for different variables, which would enable 
support to be more effective. One example was the older student who took a long time to feel 
comfortable in class. Without a strong motivation to complete, students who don’t feel 
comfortable leave. 

Finally, the survey tool may have the potential to improve class participation simply by encouraging 
students to reflect on their own performance, as indicated both by comments from students and 
the improved participation in classes where students were given feedback on the survey results. 

Student success is the result of a complex set of factors. For class-based delivery, the survey tool 
was shown to be a useful addition to other class-management techniques. It has the potential to 
both identify students with a ‘high risk’ profile early in the semester and those where a combination 
of factors is making it difficult to keep up. Finally, the measures tracked in the survey can be used 
to inform intervention strategies. 

This survey did not consider success rates for students who elect to study by other than class-based 
attendance. The literature and my experience suggest that, although students who drop out indicate 
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that they need more flexible arrangements, success rates for these students are even lower. There is 
a need for more study in this area. This research specifically excluded such students; however, the 
survey tool could be even more useful where students are isolated from class interaction. 

In summary: 

•y" It is possible to obtain useful feedback about student progress in the first weeks of class. 

The overall score based on all measures was a useful predictor of success. 

•y" Student success was based on a complex set of factors. 

Leaving a course could be linked to a combination of the factors considered in the survey. 

•y" Attendance is a critical factor in module completion. 

Students with a strong career focus appeared to have better attendance and module completion, 
although the converse was not necessarily true. 

Implications 

Students will always drop out of courses, regardless of how perfect our systems are. Over time, 
drop-out has been attributed to students, colleges and courses. As Uren pointed out in 2001, all 
three factors contribute. This project has suggested some issues that may help retain more students, 
especially those who just ‘stop coming’. This research has identified issues at a number of levels 
that can be addressed. 

1 Provide better feedback : this survey has shown that in many cases there are indicators that a student 
is likely to drop out. An adaptation of this survey could be used to obtain information from 
students before they disappear from class. Responses need to be collated quickly and the results 
made available to students in a timely manner. Obvious methods are online surveys, possibly 
used with student emails. 

2 Provide better career information-, this should be done not only before class but throughout the 
semester, especially when students begin to question if the course is right for them. Students are 
often better placed to start a different course if they have at least partly completed one course. 

3 Provide better course information-, this may involve providing ‘taster’ courses for school leavers and 
‘transition’ students as well as course-information sessions, which allow students to make 
realistic decisions. 

4 Provide support that allows students to recover after missing one or two classes-, this may be having the notes 
available online, or just spending part of each class checking whether all students at least have 
the material from the week before. 

5 Set up procedures to follow up students who miss classes-, mentoring programs where students who do 
miss class are contacted by phone or email have been shown to have some success. Students 
often do not know what options are available for help if they are feeling overwhelmed or need 
support. 

6 Demonstrate strong links with industry-. Misko and Priest (2009, p.l 1) reported that students expect 
their teachers to have a first-hand knowledge of the industry they represent. Links can be 
developed by establishing work experience opportunities, inviting industry speakers and valuing 
part-time teachers who work in their industry areas. 

7 Let class-based students know that attending class is important while there are a growing number of 
options for non-class-based delivery, students who elect to attend class should be aware of the 
importance of participation and attendance. 
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8 Provide orientation programs that help students-, this is especially important to success for students who 
fall into known high-risk categories. Institutes often have support programs that are not used by 
students because they are either not aware of them or they do not ask for help until it is too late. 

9 Provide options for students ivho want specific skills: there is much evidence that some students just 
want skills rather than a qualification. Unit-based enrolment (such as selected studies) may allow 
these students to learn the skills they need, without their having to attempt units they have no 
interest in. 

The correlation between measures for different classes may be useful in determining issues that 
affect student outcomes. For example, it has been noted that there was a correlation between ‘help’ 
and ‘understanding’, in that students who received enough help had good understanding. When the 
results were separated by class, however, the correlation held up for two classes, but not the third. 

A possible explanation for this was that, no matter how much help a student was getting, they still 
didn’t understand. In other words, the student may have been in the wrong course. 



Continuing research 

The pilot study showed that, by adopting the tool used in this project, we can gain useful 
information about students who leave. Further development of the survey tool would include 
malting it more accessible, such that data are automatically collated and feedback is provided to 
students. This could be done if the system were transferred to an electronic format. The original 
access database prototype included this functionality, but proved too difficult for students without 
good computer skills to access regularly. In addition, students need to be guaranteed that their data 
will be secure and not accessed by other students. 

Many issues were identified by this study. With a more representative sample of data, better 
statistical analysis can be applied. As an example, it is likely that many of the issues discussed are 
more relevant to young people with poor educational backgrounds (as identified in previous studies 
such as John [2001] and Uren [2001]). The sample group was not large enough to identify trends 
with smaller groups of students. 

There will not be one solution to this problem, just as there is not one single issue causing students 
to drop out. This semester we are extending the survey to a larger group of students. As issues are 
identified, we need to develop ways to address these issues and to assess the methodology used for 
this process. There is a great deal more research needed in this area. 
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Appendix A: Defining completion 



There can be variations in what is counted as completion and in the way enrolments are reported 
(Polesel, Davies & Teese 2004; Uren 2001). Even module completion is not a uniform measure; for 
example, modules vary greatly in size (or nominal hours to complete). Both ‘load pass rate’ and 
‘module load completion rate’ are used to account for differences in size of modules. For this 
report, module-completion rate is defined as the number of pass results against the number of pass, 
fail and withdrawn results. Multi-year, tuition and no-start results are not included. 

Module-completion rate = pass / (pass + fail + withdrawn) 

In contrast, course-completion rates compare qualifications awarded against commencements. For 
vocational education, completion rates tend to be much lower than module-completion rates. 
Karmel and Virk (2006) noted an overall completion rate of 57% for apprentices in 2002—05. 

Grant (2002) noted a much lower rate of 35% for New South Wales. Foyster, Fai and Shah (2000) 
described the difficulties in calculating completion rates for vocational education courses and 
estimated an overall completion rate of just 27%. 

Both measures are obtained from statistical data collected nationally and are widely used to 
compare performance over time, course and institution. They can be matched with demographic 
data to understand, on a large scale, what influences student success. Information obtained from 
projects such as Grant (2002), Polesel, Davies and Teese (2004) and Shah and Burke (2003) was 
used as a foundation for this research. 

Many authors have shown that course completion may not be the best indicator of student success 
in vocational education. Factors include the high rate of part-time attendance and employment, so 
that students often take longer to complete a qualification (Kirkby 2000). Students can move 
between courses (and into higher-level courses) without completion. Students transfer between 
institutions and courses. It has been argued that even part-completion of a qualification is useful, 
and students who do not complete a qualification may have gained what they set out to learn 
(Foyster, Fai & Shah 2000). In addition, vocational education in Australia is based largely on 
training packages and is thus structured as units of competency-based skills. Students may complete 
the competencies they need and then leave the course. Shah and Burke (2003) identify these 
students as ‘partial completers’. Given these factors, completion of units is a better indication of 
success than completion of courses. One exception may be trainees and apprentices, where there is 
a well-defined start and finish date. 

Module-completion rates also do not tell the full story. A report by Trevaskis (2009) noted that 
administrative issues influence module-completion rates. Local policies such as class unit groupings, 
delivery patterns, and enrolment and assessment procedures influence module-completion rates. 

For example, a teacher may deliver a set of units sequentially or integrated over a semester. 
Integrated delivery may result in higher withdrawn rates than sequential delivery if students are 
enrolled in all units for the semester, rather than progressively enrolled in units as they are taught. 
Fail and withdrawn rates are both counted as non-completions, but occur for very different 
reasons. Furthermore, many authors have commented on the tendency of VET students to take 
time to complete a qualification (Callan 2005; Kirkby 2000; Shah & Burke 2003). Students drop 
out, return, miss a semester, change courses, get jobs, and sometimes eventually complete. Despite 
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these considerations, and regardless of the measure that is used, we know that students drop out of 
courses that they intend to complete. 

For teachers, a more practical measure of student success is participation that continues until the 
end of the planned delivery time. Most teachers have experienced the frustration of losing a student 
part-way through a course or semester. In many cases, the perception is that the student would 
have succeeded if only they had persisted. As many researchers have reported, these students often 
return and many do eventually succeed (Callan 2005; Foyster, Fai & Shah 2000; Grant 2002; Kirkby 
2000; Mclnnis et al. 2000; Polesel, Davies & Teese 2004). 
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Appendix B: Surveys 



Initial survey 



Enter your username OR full name: 

First Name Last Name 




What is the name of this course? 
What is your current age? 



What is your gender? Female 



What Language do you speak at home? English 



Which year did you finish school? Year 12 



Where did you complete your school education? Australia 



What is your highest post school education? Certificate 3 
I have no idea what career 

or job I want to follow I I 



I am doing this course 
mostly for personal 
interest 
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Survey questions — weekly survey 



Looking back over the last week, help us Marks to the left represent low levels and marks 

understand how you have been feeling about your to the right indicate high levels. 

course, by rating these different aspects. Click on the line where it feels right for you. 



I don't want to 
complete this today 



click on the bar to respond 



Date: 06-Apr-10 



I liked nothing in this 
course in the last week 

I understood nothing in 
this course in the last week 



I have not attended any 
classes this week 

I have done no work on 
assignments this week 

Life is too busy or 
demanding this week 



I am not getting any help 
from my teachers 

I am not at all 
comfortable in my classes 

I do not think I will 
complete this course 



I liked everything in this 
course in the last week 

I understood everything in 
this course in the last week 

I have attended all of my 
classes this week 

I have done sufficient work 
on my assignments this week 

Life is easy this week 

I am getting enough help 
from my teachers 

I feel completely 
comfortable in my classes 

I am sure I will complete 
this course 



Save and Close 



Comment: 



Questions asked of surveyed students: 

Did you find any of the survey questions difficult to answer? 

Were you able to give accurate responses? 

Did answering the survey have any effect on your involvement with the course? 

Were there any questions that were ambiguous; that is, not clear in their meaning? 

What happened in week . . . (this question was related to specific features that were noted in the 
charts for particular students or student groups). 

Was there any other information that would have liked to include? 
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Appendix C: 
Comments from student surveys 



Comments 

This week has been hard. I have gone thru a court case and life is mentally insaner, but I’m still 
coming to TAFE because I don’t want to fall behind. I want to move out asap and get a good 
job, and I need to get TAFE done so I can get a good job. 

This week life has been busy, and I’m having troubles with learning my calculator for stats. My 
teacher is not here today and I feel that I will fall behind. 

Think Positive :) 

Tafe is very flexible and provides a better school environment than High School. 

Some days I don’t think I will get through it all. 

Seem to be falling behind a little bit — should be able to make it up in following weeks. 

My stat teacher was sick. I’m dropping class. 

My comfortability within the classroom has definitely eased over the last 6 weeks. I would like to 
express that I was very uncomfortable initially. I have struggled with the ‘casualness’ of fellow 
classmates eg constant cigarette breaks, ph calls, txt mess. 

I was off sick and with other family commitments the past two weeks. 

I started a new job this week so my priorities have been mixed up and my home work has got 
piled up. 

I only attend the Statistics short course, Communication, and LALS classes this semester as I 
work in the afternoons. 

I love TAFE, just this week has been difficult due to personal struggle, so it’s affecting school, 
but I’m coping. 

I have had the previous week off whilst on holidays and now I feel swarmed with work over my 
head, and for some reason don’t know why I can’t understand what it is I have to do. But I have 
this constant knowing in my head that I am behind. SO ANNOYING, :) 

I find my Friday Class lacks structure. 

I do not like public speaking, therefore do not like comms and LALS that much. 

Has been a busy few weeks for me what with some unexpected work and a few personal issues 
that have eaten a lot of my time up. Overall though, I am comfortable with the course so far and 
am really enjoying learning more and more. 

Great support! Thank you! 

Generally very happy with the teachers and the content of the course. 
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Flu is going on at my family and that made my life a bit harder than usual week. 

Feeling much more comfortable within the course now. Class more at ease. 

English is always hard. My teachers are fabulous. I’ve been sick this week. I’m struggling with the 
work load. 

Best place to get educated. 
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